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(g) A display system for visualization of body structures during medical procedures. 

(§)' An interactive display system superimposes 
radiological images on a semi-transparent 
screen through which a surgeon views a patient 
during a medical procedure. The superimposed 
image is derived from image data obtained with 
an imaging system. The radiological image is 
registered with the surgeon's view of the patient 
and displayed in real-time during a medical 
procedure. This allows the surgeon to view 
internal and external structures and the relation 
between them simultaneously, and adjust the 
procedure accordingly. A second embodiment 
employs stereoscopic viewing methods to pro- : 
vide three-dimensional representations of the 
radiological images superimposed on the 
semi-transparent screen through which the sur- 
geon views the patient. 
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BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a system for aid- 
ing a medical practitioner, such as a surgeon, in vis- 
ualizing anatomical structures during medical proce- 
dures and more specifically to a system which allows 
the practitioner to view, in real time > both internal and 
external anatomical structures. 

2. Discussion of Prior Art 

This application is related to International appli- 
cation US 94/02572 "Computer Graphic and Live Vid- 
eo System for Enhancing Visualization of Body Struc- 
tures During Surgery" and to European Patent appli- 
cations 0549183 "System For Displaying Solid Cuts 
For Surfaces of Solid Models", 0549189 "Solid Mod- 
els Generation By Span Method Using Dividing 
Cubes" and 0549182 "Apparatus and Method For 
Displaying Surgical Cuts in Three-Dimensionai Mod- 
els". 

Presently, during surgery and other medical pro- 
cedures such as endoscopy, biopsy and implantation, 
physicians view several static radiographic views of 
the patient in the operating room. Typically these are 
transparent film renderings of magnetic resonance 
(MR), computed tomography (CT), conventional X- 
ray images or ultrasound images. Since these im- 
ages are two dimensional static images, the physi- 
cians must determine actual three-dimensional (3D) 
location and shape of desired internal structures 
within the patient from the 2D images which they are 
viewing. Conceptually, the surgeon constructs a 3D 
model of the internal structures and correlates these 
internal structures with visible external structures of 
the patient where they must cut This is often difficult 
because the scale and the orientation of the 2D image 
may differ from what the surgeon is seeing, and the 
surgeon may not be able to view both the patient and 
the medical diagnostic images simultaneously. 

Another technique employed in localization of in- 
ternal structures during surgery is known as stereo- 
tactic surgery as described in "Interactive Stereotac- 
tic Surgical System for the Removal of Intracranial Tu- 
mors Utilizing the CQ2 Laser and CT-Derived Data- 
base" by B. A. Kail, P. J. Kelly, and S. J. Goerss, IEEE 
Transactions on Biomedical Engineering, vol. BME- 
32, no. 2, pp 112-116, 1985; and "Comprehensive 
Computer-Assisted Data Collection Treatment Plan- 
ning and Interactive Surgery" by B. A. Kail, P. J Kelly, 
and S. J. Goerss, Medical Imaging, vol. 767 pp. 509- 
514, 1987. With this approach, a rigid mechanical 
frame is attached to the patient before a CT or MR 
procedure. The frame and its landmarks can be seen 
in the resulting images. Mechanisms on the frame 
position a probe at specific location within the image. 



The disadvantages of this approach are that the 
frame limits access to the patient, and the images are. 
static images which do not follow the patient if he 
moves during surgery. 

A third technique used for localization of internal 
structures is described in "A Frameless Stereotaxic 
Operating Microscope for Neurosurgery" by E. M. 
Friets, J. W. Strohbehn, J. F. Hatch, and D. W. Ro- 
berts, IEEE Transactions on Biomedical Engineering, 
vol. 36., no. 6, pp 606-617, June 1989. 

Three dimensional models of anatomical struc- 
tures can be created from data of different medical 
imaging modalities as described in the applications 
listed above. These applications describe creating 
and manipulating models of internal structures of pa- 
tients and providing images of selected structures at 
desired orientations to an operator. These allow visu- 
alization of internal structures as solid models. 

Currently there is a need for a system to aid 
physicians in surgery and other medical procedures 
which interactively displays computer generated rep- 
resentations of internal structures in correct relation 
with external structures of the patient. 

SUMMARY OF THE INVENTION 

A feature of the present invention is to provide a 
system which aids in surgery by simultaneously su- 
perimposing an image of internal structures upon ex- 
ternal structures even if the patients changes their 
position or if the operators change their viewing an- 
gles. 

Another feature of the present invention is to pro- 
vide an interactive system which displays desired in- 
ternal structures upon external structures, having 
the same scale and viewed from the same orientation 
angles. 

Another features of the present invention is to 
provide a surgeon a "heads up" display of radiological 
images superimposed upon a patient within his field 
of view. 

A real-time surgery apparatus for displaying in- 
teractive internal and external images of a patient 
employs a semi-transparent display screen allowing 
an operator to simultaneously view exposed surfaces 
and computer generated images of a patient 

A medical imaging device obtains three- 
dimensional (3D) imaging data of internal structures 
of the patient which are fed to a workstation. The 
workstation creates three-dimensional (3D) comput- 
er generated models which may be manipulated with- 
out further need for the medical imaging device. Two- 
dimensional computer generated images of the mod- 
els are interactively oriented and scaled so that their 
display on a semi-transparent display device coin- 
cides with the operator's visual image of the patient. 

The three-dimensional position and orientation of 
the patient, display device and operator's eyes are 
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monitored in real-time. This information is then used 
to adapt the computer generated display on the semi- 
transparent screen so that the visual and computer 
generated images consistently coincide. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The features of the invention believed to be novel 
are set forth with particularity in the appended claims. 
The invention itself, however, both as to organization 
and method of operation, together with further ob- 
jects and advantages thereof, may best be under- 
stood by reference to the following description taken 
in conjunction with the accompanying drawing in 
which: 

Fig. 1 is a simplified block diagram of a frst em- 
bodiment of a medical display apparatus according to 
the present invention. 

Fig. 2 is a simplified block diagram of a second 
embodiment of a medical display apparatus accord- 
ing to the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

In Fig. 1, a patient 1 on which a medical proce- 
dure such as surgery is to be performed, is scanned 
by a medical imaging apparatus 10 which may be a 
magnetic resonance (MR) imaging apparatus, a com- 
puted axial tomography (CAT) apparatus, a positron 
emission tomography (PET) or similar imaging device 
capable of creating multi-dimensional volumetric 
data such as 3-dimensional (3-to) data, from internal 
structures of the patient After imaging, apparatus 10 
provides the volumetric data to a model workstation 
100. Once the volumetric data has been provided to 
model workstation 100, further need for imaging ap- 
paratus 10 imaging apparatus is no longer required 
This js important since some medical procedures 
need not be performed with the patient situated within 
the confines of an imaging apparatus, which can be 
constricting in the case of MR imaging. In alternative 
embodiments, imaging apparatus 10 may be interac- 
tively employed during the medical procedure. Model 
workstation 100 stores the volumetric data and cre- 
ates computer generated models from the data capa- 
ble of being seated, rotated and otherwise manipulat- 
ed, without the further need for imaging apparatus 
10. ' 

An operator 350, such as a physician or medical 
assistant, monitors patient 1. A semi-transparent 
screen 250 rs interposed between patient 1 and op- 
erator 350. A tracking device 50 which monitors and 
tracks operator 350, semi-transparent screen 250 
and patient 1 determines a relative roll a, pitch e, and 
yaw f orientation between operator 350, semi- 
transparent screen 250 and subject 1 . Tracking de- 
vice 50 may be a 6-degrees of freedom tracking de- 
vice as described The Flock of Birds' Installation and 



Operation Guide, Ascension Technology Corporation, 
July 5, 1992, in the Introduction pp. 1-3, Appendix 1, " 
p. 89 and Appendix 3, p. 93. Tracking device 50 also 
determines a location (in Cartesian coordinates) of 
operator 350 and semi-transparent screen 250 with 
relation to patient 1. Patient 1 is assumed to be at the 
origin of the Cartesian coordinate system (x,y,z) = 
(0,0,0), therefore all distances relative to the patient 
are simpfy the (x, y,z) location. The location and ori- 
entation are interactively provided to model worksta- 
tion 100 by tracking device 50. The location and ori- 
entation may also be provided manually to model 
workstation(s) in different embodiments. 

Model workstation 100 processes the 3D volu- 
metric data it receives and creates selected render- 
ings of the data. One rendering method determines 
surfaces between differing types of tissue. Connec- 
tivity of similar types of tissue adjacent to one another 
is then determined. Differentiation of tissue types 
based on the nature of the signal in the three- 
dimensional image data is known as segmentation. 
When the 3-D volumetric data has been segmented 
into internal structures, each internal structure may 
be treated as a separate solid object by model work- 
station 100. The model workstation has the capability 
of selectively displaying desired internal structures, 
color coding structures and severing, rotating and 
translating internal structures in order to manipulate 
these images in a desired manner to provide visuali- 
zation to an operator working model workstation 100. 

An alternative rendering method generates two- 
dimensional projections of selected features within 
the three-dimensional data set is described in Euro- 
pean Patent Application 0506302 'Projection Meth- 
ods for Producing Two-Dimensional images from 
Three-Dimensional Data". For example, two-dimen- 
sional projection angiograms can be extracted from a 
three-dimensional phase contrast or time-of-f light 
magnetic resonance angiogram Several projection 
algorithms are possible. These include the detection 
of the maximum pixel intensity along a selected pro- 
jection ray through the three-dimensional data, deter- 
mination of the average pixel intensity of a selected 
projection ray and the determination of the standard 
deviation of all pixels along a selected projection ray. 

Model workstation 1 00 receives in put data from 
a model cut plane input device 40 and a workstation 
view input device 60 to select the method of display- 
ing internal structures of patient 1. Model cut plane in- 
put device 40 and a workstation view input device 60 
may be a computer pointing device such as a mouse 
or trackball, or any input device which indicates 
planes in which to cut the images and a viewing angle 
and scale. Tracking device 50 provides relative orien- 
tation data between operator 350 and patient 1 and 
scaling, which allows model workstation 100 to syn- 
thesize an interactive computer generated image of 
internal structures of patient 1 and have it coincide 



10 



15 



20 



25 



30 



35 



40 



45 



50 



5 



EP 0 629 963 A2 



with the real-time scene viewed by operator 350. This 
interactive computer generated image may be con- 
verted from a computer monitor signal, such as an 
RGB computer monitor signal, to a video format by 
passing it through a scan converter 192 (shown in 5 
phantom), depending on the display format of semi- 
transparent screen 250. The computer generated im- 
age is provided to semi-trans parent screen 250 which 
is interposed between patient 1 and operator 350. 
Semi-transparent screen 250 provides a desired mix* 10 
ture of external structures of patient 1 seen by oper- 
ator 350 and the computer generated image from 
model workstation 100. Semi-transparent screen 250 
may receive input signals from the operator, for exam- 
ple, through workstation view input 60. This may in- 15 
volve the degree of transparency of each image, or 
any of various other special effects. One very useful 
special effect is a movable window which has 0% 
transparency (100% opaqueness) superimposed 
upon another image. When the window image is of in- 20 
ternai structures superimposed upon external struc- 
tures, it creates the illusion of external structures cut 
away within the window, exposing underlying interna! 
structures. Other conventional video special effects 
may also be employed. Semi-transparent screen 250 25 
allows operator 350 to visual tze both internal and ex- 
ternal structures simultaneously. The resulting scene 
witnessed by operator 350 is an interactive real-time 
image of patient 1 , even if the patient moves during 
the medical procedure. Since internal structures and 30 
their relation to exposed external structures are si- 
multaneously displayed, a surgeon using the present 
invention will perceive a very accurate indication of 
where he should cut through external struct ures to ar- 
rive at a desired internal structure while avoiding vital 35 
internal structures. 

Figure 2 illustrates an alternative embodiment of 
the present invention in which operator 350 receives 
a stereoscopic view of internal and external struc- 
tures of patient 1. Each of the eyes of operator 350 46 
differs in its orientation with relation to the patient 
therefore a different view is provided to each eye. 
Tracking device 50 tracks the relative location 
(x lt y l ,z 1 ) and orientation angle (a tr fi, 80 between a 
first eye of operator 350 and patient 1. Tracking de- 45 
vice 50 also tracks a second location (X2,y 2 ,z 2 ) and 
orientation (a^fc.Oa) between a second eye of oper- 
ator 350 and patient 1. The second location and ori- 
entation can be measured independently by tracking 
device 50; or the location and orientation of the first so 
eye can be used to compute the location and orienta- 
tion of the second eye^ The locations and orientations 
are fed to a first model workstation 100a, and a sec- 
ond model workstation 100b which create a right and 
left computer graphic image at locations (x^y^), 55 
(x2.y2.Z2)> respectively, and orientations (a t , fo, 0-0 
(a 2 .fc» 0 2). respectively, corresponding to the views of 
each eye of operator 350, respectively. Control sig- 



nals from model cut plane 40, workstation view input 
60 and imaging apparatus 10 can be sent to both first 
model workstation 100a, which in turn propagates the 
control signals to second model workstation 1Q0b via 
communications path 140, or alternatively, the con- 
trol signals can be sent directly to both model work- 
stations directly. 

The left and right computer generated image, 
pertaining to a left and right view, respectively, are 
converted to video format if required by semi- 
t ransparent scree n 250. This is accomplished by scan 
converters 192a, 192b (shown in phantom) which 
pass the converted computer generated signals to a 
sequencer 1 98. Sequencer 1 98 may be a convention- 
al video sequencer as described in Tortable, Low 
Cost Devices for Videotaping, Editing and Displaying 
Field Sequential Stereoscopic Motion Pictures and 
Video" by M. Starks, Stereoscopic Displays and Ap- 
plications Proc. SPIE Vol. 1256, pp. 266-271, 1990. 

Sequencer 198 passes the left computer gener- 
ated image to semi-transparent screen 250. Se- 
quencer 198 then passes the right computer generat- 
ed image to semi-transparent screen 250. Sequencer 
198 alternates many times per second, in synchroni- 
zation, between right and left views. 

The image displayed on semi-transparent screen 
250 is time multiplexed to produce an image to the left 
eye and right eye of the operator in an alternating 
fashion. A stereoscopic viewer 252 is synchronized to 
a sequencer 1 98 and operates to block the vision of 
the operator's left or right eye allowing the opposite 
eye to view the image on semi-transparent screen 
250 for an instant and vice-versa. This allows opera- 
tor 350 to see the left image with the left eye while the 
right eye sees nothing and the right image with the 
right eye while the left eye sees nothing in rapid suc- 
cession. This creates a stereoscopic illusion, adding 
the dimension of depth perception in viewing the im- 
age displayed on semi-transparent screen 250. 
Depth perception is very valuable in surgery since it 
adds a dimension that assists in visually localizing 
structures, which is especially important in complex, 
delicate surgery. 

For both embodiments of the invention the com- 

; puter image must be registered (coincide) with the ex- 
ternal structures as viewed by operator 350. Initiali- 
zation may be accomplished by manual input from the 
operator to rotate, translate and scale the computer 
generated image(s) until they coincide with the scene 
observed through the semi-transparent screen, or by 
employing tracking device 50 to set initial parameters. 

Once the 3D model and the visual image of pa- 
tient tare aligned, tracking device 50 keeps the view 
angles and-f ield of view consistent. This allows real- 
time interactive synchronization between the opera- 
tor's view of patient 1 and the computer generated tm- 
age(s). 

Since the image of each internal structure can be 
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segmented into what resembles a solid object, it may 
be manipulated as a solid object In the case of struc-. 
tures of a patient, a surgeon may acquire data f rom 
the patient by medical imaging, then plan surgery by 
manipulating the models to plan a desired result be- 5 
. fore surgery. This is common in complex reconstruc- 
tive surgery. Once the plan is determined, it may be 
stored and played back during surgery. The Images 
of internal structures are interactively oriented and 
scaled to coincide with the actual patient to 

A user employs as a model cut plane input device, 
a workstation view input device (40, 60 of Figs. 1 and 
2, respectively) to select planes in which to cut the 
' structures in the model, to select a three-dimensional 
orientation of the model, and to select screen cut 1 s 
planes which define a workstation viewing region. 
The model workstation can incorporate a clipping cir- 
cuit to determine points within the model cut planes, 
a rotation circuit to rotate points and normal vectors, 
a segmentation processor, and a shading circuit 20 
which determines shading based upon the orientation 
of the normal vector at each point In addition a screen 
clipping circuit can be used to determine points within 
a region defined by the screen cut planes. The model 
workstation also can include a display circuit to create 25 
video signals which, when propagated to a suitable 
display device, generate images of multiple surfaces 
that are within the desired display region and the 
screen cut planes. 

In radical surgery such as ablative surgery, or 30 
massive trauma cases/there is little structure which 
remains to correctly determine what a normal anato- 
my should be. In these cases, an additional model 
workstation may have a model of normal structures 
stored which may be mixed with the other images be- 35 
ing displayed to act as a guide in reconstructive sur- 
gery. This may be implemented by additional work- 
stations or model manipulation boards. 

In the present embodiments of the invention, 
semi-transparent screen 250 is interposed between 40 
operator 350 and patient 1. Screen 250 can be con- 
structed with a liquid crystal display or it can be com- 
prised of a partially silvered mirror reflecting an image 
• from a video monitor. Semi-transparent display 250 
can be constructed as a relatively large device having 45 
dimensions approximately equal to that of the region 
of interest of patient 1, or alternatively it can be of 
small dimension and placed relatively close to the op- 
erators eyes, perhaps incorporated into headgear or 
eyewear. , 50 

While several presently preferred embodiments 
of the novel visualization system have been descri- 
. bed in detail herein, many modif ications and varia- 
tions will now become apparent to those skilled in the 
art It is, therefore, to be understood that the append- 55 
ed claims are intended to cover all such modifications 
and variations 



Claims' . 

1. A real-time medical apparatus for displaying to an 
operator interactive images of internal structures 
of a patient coihcidihg with a view by the operator 
of the patient comprising: . 

a) a medical imaging system for obtaining 
multt-dirrtenslonal imaging data of internal 
structures of said patient; 

b) a workstation for creating images from the 
imaging data of internal structures of said pa- 
tient viewed from a location (x,y,z) and orien- 
tation (a f $,8) which coincide with said view of 
the patient by said operator; 

c) a semi-transparent screen adapted for al- 
lowing said operator to view said patient 
through the semi-transparent screen and 
adapted for displaying the images of internal 
structures from the workstation with a desired 
degree of transparency to create the illusion 
of internal structures superimposed upon 
said patient 

2. The real-time medical apparatus of claim 1 fur- 
ther comprising a tracking device adapted to 
measure the location (x,y,z) and orientation 
(a,$,9) of said operator with respect to said pa- 
tient for repeatedly providing these measure- 
ments to the workstation. 

3. The real-time medical apparatus of claim 1 
wherein the workstation further comprises surgi- 
cal planning means adapted to create models of 
internal structures of said patient, interactively 
manipulate the models to result in modifications 
to internal structures, store the models and mod- 
ifications and display the models and modifica- 
tions viewed f rom a location and orientation to co- 
incide with the view of the patient by the operator. 

4. The real-time medical apparatus of claim 1 
wherein the workstation incorporates normal 
anatomical models of internal structures, and is 
adapted to display the models viewed from a lo- 
cation and orientation to coincide with the view of 
the patient by the operator, such that said work- 
station can act as a guide in reconstructive sur- 

. . gery.- -r;-"' *'\->vV ; ' . 

5. A real-time medical apparatus for displaying to an 
operator interactive i three-dimensional (3D) inter- 
nal and external images of a patient comprising: 

a) a medical imaging system for obtaining 
three-dimensional (3D) imaging data of inter- 
nal structures of said patient; 

b) tracking means for measuring locations 
(x^y^t) (xi t y 2 ,Z2) and orientation angles 

(a^fc.OJ of each eye of said opera- 
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tor with respect to said patient; 

c) a first workstation for creating a right com- 
puter generated image of internal structures 
of said patient from the 3D imaging data from 

the medical imaging system as viewed from 5 
the location (x^y^) and orientation angle 
(a 1t ti,0i) obtained from the tracking means; 

d) a second workstation for creating a left 
computer generated image of internal struc- 
tures of said patient from the 3D imaging data to 
from the medical imaging system as viewed 
from a location (x^y^J and on^ntation angle 
(a 2 ,^ 2 ,02) obtained from the tracking means; 

e) a semi- trans parent screen adapted to allow 
said operator to view said patient through the f 5 
semi-transparent screen and adapted to dis- 
play the left and right computer generated Im- 
ages of internal structures from the worksta- 
tions with a desired degree of transparency in 
rapid succession; 20 

f) stereoscopic display means synchronized 
with the semi-transparent screen for allowing 
only the operator's right eye to view the semi- 
transparent screen when the right computer 
generated image is being displayed, and for 25 
allowing only the operator's left eye to view 

the semi-transparent screen when the left 
computer generated image is being displayed 
thereby simulating a 3D image of internal and 
external structures of said patient 30 

6. A method of aiding an operator to perform a med- 
ical procedure upon a patient comprising the 
steps of: 

a) acquiring multi-dimensional medical imag- 35. 
ing data from internal structures of said pa- 
tient; 

b) measuring a location (x,y,z) and orientation 
angle (a t $,G) of said patient as viewed by said 
operator, *o 

c) creating a computer generated image of the 
internal structures from the medical imaging 
data viewed from the location (x,y,z) and ori- 
entation angle (a^,8); 

d) interposing a semi-transparent screen be- 45 
tween said operator and said patient allowing 

said operator to view said patient through the 
semi-transparent screen; and 

e) displaying the computer generated image 

on the semi-transparent screen with a desired so 
degree of transparency to create the illusion 
of internal structures superimposed upon 
said patient to assist the operator in said med- 
ical procedure. 

55 

7. The method of aiding an operator in a medical 
procedure of claim 6 wherein the step of acqu iring 
medical imaging data is performed in real-time by 



a medical imaging system. 

8. The method of aiding an operator in a medical 
procedure of claim 6 further comprising the step 
of creating a computer generated guide image of 
normal anatomical internal structures viewed 
from a location (x,y,z) and orientation angle 
(a,4>,9) to be used as a guide to reconstructive 
surgery; and mixing portions of the guide image 
with the computer generated images. 

9. A method of aiding an operator in a medical pro- 
cedure of a patient comprising the steps of: 

a) obtaining three-dimensional (3D) imaging 
data of internal structures of said patient; 

b) interposing a semi-transparent screen be- 
tween said operator and said patient allowing 
said operator to view said patient through the 
semi-transparent screen; 

c) measuring the locations (x^y^) (x 2 ,y 2 ^2) 
and orientation angles (a t ,+i ,9^ (a 2 ,$2*fy) be- 
tween said patient and each eye of said oper- 
ator, respectively, 

d) creating a left computer generated image 
and a right computer generated image of in- 
ternal structures from the 3D imaging data of 
said patient viewed from distances (x 1 ,y 1 ,z 1 ) 
(x 2 ,y 2 ^ 2 ) and orientation angles (ai,$ f ,0i) 
(v-iMf^d respectively; 

e) displaying the left and right computer gen- 
erated images of internal structures of a de- 
sired degree of transparency in rapid succes- 
sion on the semi-transparent screen; and 

f) blocking the view of the left eye of the op- 
erator when the right computer generated im- 
age is being displayed, and blocking the view 
of the right eye of the operator when the left 
computer generated image is being displayed 
thereby simulating a 3D image of internal and 
external structures of said patient 

10. The method of aiding an operator in a medical 
procedure of claim 9 further comprising the step 
of adjusting the distances (x^y^Zt) fey^z^ and 
orientation angles {v.\M#\) (a2.te.G2) of the ^ ert 
and right computer generated images to corre- 
spond to the view of the patient by the operator. 
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